The present invention relates to image synthesizing apparatus and method, position detecting apparatus and method, and a supply medium, particularly to image synthesizing apparatus and method, position detecting apparatus and method, and a supply medium for making it possible that the projected image of a predetermined object seems to be natural in continuous photographic images by synthesizing the projected image of the predetermined object with a plurality of continuous photographic images.
Because the computer graphics (CG) art has been advanced in recent years, it is possible to form a real image almost same as an actually photographed image (taken on the spot) though;the real image is artificially formed. In this case, the processing is frequently performed in which a predetermined object is continuously photographed (as a dynamic image) by a video camera and then, the CG image of the predetermined object is synthesized every image taken on the spot.
FIG. 1 shows a conventional image synthesizing method. In this case, it is assumed that an existent plate 1 is picked up by an image pickup section 2 and balls 5 and 6 formed through CG are synthesized with the image of the plate 1 so that the balls 5 and 6 seem to be stationary. In this case, the plate 1 on which patterns are drawn is first actually picked up by an image pickup section 2. In this case, the i-th pattern formed on the plate 1 is assumed as ai. In this case, symbol i denotes an integer and also an identification number attached to each pattern ai. The patterns ai are different from each other in color and shape and therefore, they can be distinguished between them. The image pickup section 2 uses a stereophonic video camera constituted with a main video camera and a sub-video camera and is constituted so that three-dimensional information can be obtained from the parallax between an image picked up by the main video camera and an image picked up by the sub-video camera. The plate 1 is picked up while moving the image pickup section 2.
In this case, the time for the image pickup section 2 to start pickup is assumed as a first time and a predetermined time after the first time in the pickup time is assumed as a second time. FIG. 1 shows the image pickup section 2 at the first time and the second time.
As shown in FIG. 2, a frame image 1A1 obtained by picking up the plate 1 from a predetermined angle is obtained from the image pickup section 2 at the first time. Similarly, as shown in FIG. 3, a frame image 1A2 different from the frame image 1A1 obtained at the first time is obtained at the second time.
Then, as shown in FIG. 4, projected images 5A1 and 6B1 of the balls 5 and 6 are synthesized with the image (image in FIG. 2) 1A1 picked up by the image pickup section 2 at the first time. Then, as shown in FIG. 5, projected images 5A2 and 6B2 of the balls 5 and 6 are synthesized with the image (image in FIG. 3) 1A2 picked up by the image pickup section 2 at (the second time at positions where the balls 5 and 6 seem to be stationary on the plate 1.
In this case, a method of synthesizing the balls 5 and 6 formed through CG so that they seem to be stationary on the plate 1 is described below by referring to FIGS. 6 and 7.
Though positions for finally synthesizing projected images 5A and 6B are on the two-dimensional images 1A1 and 1A2, the synthesis positions are easily understood by considering them in a three-dimensional space. Therefore, three-dimensional data (coordinates) is restored from a two-dimensional frame image. Because the frame image is obtained by the stereophonic video camera, the above restoration is realized.
That is, as shown in FIG. 6, three-dimensional coordinates are assumed in which the position of the image pickup section 2 at the first time is an origin O1 and the horizontal, vertical, and depth directions of the image pickup section 2 are X-axis, Y-axis, and Z-axis. Then, the three-dimensional position (X1i, Y1i, Z1i) of every pattern ai of the plate 1 is first obtained from the two-dimensional frame image 1A1. Then, a user designates a three-dimensional position (three-dimensional synthesis position) (X1A, Y1A, Z1A) on which the ball 5 is put and a three-dimensional position (three-dimensional synthesis position)(X1B, Y1B, Z1B) on which the ball 6 is put.
Then, a synthesis position (two-dimensional position) on the image 1A1 corresponding to a three-dimensional synthesis position is obtained. The two-dimensional position is obtained as a position obtained through perspective projection transform of the three-dimensional position. That is, when assuming the focal distance of the image pickup section 2 as f, the two-dimensional synthesis position of a projected image 5A1 on the image 1A1 at the first time corresponding to the three-dimensional synthesis position (X1A, Y1A, Z1A) is obtained as (X1Axc3x97F/Z1A, Y1Axc3x97f/Z1A). Similarly, the two-dimensional synthesis position of a projected image 6B1 on the image 1A1 at the first time corresponding to the three-dimensional synthesis position (X1B, Y1B, Z1B) is obtained as (X1Bxc3x97f/Z1B, Y1Bxc3x97f/Z1B).
Similarly, as shown in FIG. 7, the position of the image pickup section 2 at the second time is assumed as an origin O2 and the horizontal, vertical, and depth directions of the image pickup section 2 are assumed as X-axis, Y-axis, and Z-axis. Then, the three-dimensional position (X2i, Y2i, Z2i) of every pattern ai of the plate 1 is obtained from the two-dimensional image 1A2.
As described above, because the patterns ai on the plate 1 can be distinguished between them, it is possible to identify that the i-th pattern ai in the first three-dimensional coordinate system restored from the image 1A1 at the first time corresponds to which pattern ai in the second three-dimensional system restored from the image 1A2 at the second time. Therefore, it is possible to make the coordinates (X1i, Y1i, Z1i) of the pattern ai at the first time correspond to the coordinates (X2i, Y2i, Z2i) of the pattern ai at the second time. Because these two coordinate systems view the same pattern ai from different angles, the second three-dimensional coordinate system can be obtained by applying predetermined rotational transform (hereafter, the function of the transform is assumed as R1) and predetermined rectilinear transform (hereafter, the function of the transform is assumed as S1) to the first three-dimensional coordinate system (though the function of the rectilinear transform is normally shown by T, it is shown by S because T is used as a variable showing time in this specification). Therefore, the relation shown by the following Equation is, effected for each pattern ai.
(X2i,Y2i,Z2i)=(X1i,Y1i,Z1i)xc2x7R1+S1
(Though the function of the rectilinear transform is normally shown by T, it is shown by S because T is used as a variable showing time in this specification.)
Therefore, the coordinate transform functions R1 and S1 can be obtained by substituting (X1i, Y1i, Z1i) and (Z2i, Y2i, Z2i) of each pattern ai for the above Equation.
However, when restoring the position (three-dimensional coordinates) of the pattern ai in a three-dimensional space from image data (image data for the images 1A1 and 1A2) in a two-dimensional space picked up at a certain time, the restored position includes any error. Therefore, when using, for example, the first three-dimensional coordinate system as a criterion, the position of the i-th pattern ai is not exactly present at (X1i, Y1i, Z1i). Similarly, when using the second three-dimensional coordinate system A criterion, the position of the i-th pattern ai is not exactly present at (X2i, Y2i, Z2i).
Therefore, values obtained by squaring the magnitude of the three dimensional vector (x1, y1, z1) in the following Equation (1) on every pattern ai by the least-squares method are totalized to obtain the (most-probable) transform functions R1 and S1 for minimizing the total error.
xe2x80x83(x1,y1,z1)=(X2i,Y2i,Z2i)xe2x88x92{(X1i,Y1i,Z1i)xc2x7R1+S1}xe2x80x83xe2x80x83(1)
By using the transform functions R1 and S1 obtained as described above, the three-dimensional position (X2A, Y2A, Z2A) of the ball 5 at the second time corresponding to the three-dimensional position (X1A, Y1A, Z1A) of the ball 5 at the first time is obtained by the following Equation (2).
(X2A,Y2A,Z2A)=(X1A,Y1A,Z1A)xc2x7R1+S1xe2x80x83xe2x80x83(2)
Similarly, the position (X2B, Y2B, Z2B) of the ball 6 at the second time corresponding to the three-dimensional position (X1B, Y1B, Z1B) of the ball 6 at the first time is obtained by the following Equation (3).
(X2B,Y2B,Z2B)=(X1B,Y1B,Z1B)xc2x7R1+S1xe2x80x83xe2x80x83(3)
Then, by applying the relation of the perspective projection transform to the three-dimensional coordinates (X2A, Y2A, Z2A) and (X2B, Y2B, Z2B), the coordinates (X2Axc3x97f/Z2A, Y2Axc3x97f/Z2A) of the projected image 5A2 and the coordinates (X2Bxc3x97f/Z2B, Y2Bxc3x97f/Z2B) of the projected image 6B2 on the two-dimensional image 1A2 picked up by the image pickup section 2 at the second time are obtained. By synthesizing the projected images 5A2 and 6B2 with the coordinate positions, synthesized image must be seen so that the balls 5 and 6 are stationary for the plate 1 because the coordinate positions correspond to the coordinate positions of the projected images 5A1 and 6B1.
The position of the i-th pattern ai and the positions of the balls 5 and 6 at the first time are shown as (X1i, Y1i, Z1i), (X1A, Y1A, Z1A), and (X1B, Y1B, Z1B) in the first three-dimensional coordinate system. Moreover, the position of the i-th pattern ai and the positions of the balls 5 and 6 at the second time are shown as (X2i, Y2i, Z2i), (X2A, Y2A, Z2A), and (X2B, Y2B, Z2B) in the second three-dimensional coordinate system. In this case, the relation between the,,positions of the i-th patterns ai and the positions of the balls 5 and 6 in these two coordinate systems can be shown by the following Equations (4), (5), and (6).
(X2i,Y2i,Z2i)xe2x89xa0(X1i,Y1i,Z1i)xc2x7R1+S1xe2x80x83xe2x80x83(4)
(X2A,Y2A,Z2A)=(X1A,Y1A,Z1A)xc2x7R1+S1xe2x80x83xe2x80x83(5)
xe2x80x83(X2B,Y2B,Z2B)=(X1B,Y1B,Z1B)xc2x7R1+S1xe2x80x83xe2x80x83(6)
That is, in the case of middle and bottom Equations among the above three Equations, because the left-side terms are obtained by operations of the right-side terms, the right and left terms are equal. However, as described above, the transform functions R1 and S1 are obtained by using the least-squares method and therefore, the functions include errors. As a result, as shown in the top Equation, the three-dimensional coordinates (obtained from a second image 1A2) of the pattern ai of the second image 1A2 do not accurately coincide with the coordinates obtained by applying the transform functions R1 and S1 to the three-dimensional coordinates (X1A, Y1A, Z1A) of a first image 1A1.
Therefore, the relation of projected images 5A1 and 6B1 to the pattern ai of the first image 1A1 at the first time is different from the relation of the projected images 5A2 and 6B2 to each pattern ai of the second image 1A2 at the second time. When reproducing this image string, there is a problem that the balls 5 and 6 seem to be unnatural because the balls 5 and 6 are lifted from the plate 1 on which the patterns ai are drawn.
The present invention is made to solve the above problems and its object is to provide an image synthesizing apparatus and method and a supply medium for making it possible that, when synthesizing the projected image of a predetermined object with a plurality of photographic images, a synthesized image seems to be natural by synthesizing the projected image of the predetermined object in accordance with a characteristic point nearby the synthesis position.
Moreover, it is another object of the present invention to provide position detecting apparatus and method and a supply medium capable of correctly calculating the positional relation of a third image to first and second images.
Furthermore, it is still another object of the present invention to provide image synthesizing apparatus and method and a supply medium capable of controlling the fluctuation of a synthesized image by correcting the position of the projected image of a predetermined object in accordance with a distortion value when synthesizing the projected image of the object with a plurality of photographic images.
The image synthesizing apparatus of claim 1 comprises first means for obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto a first image, second means for obtaining first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image, fourth means for obtaining coordinate transform functions of the first and second coordinates, fifth means for obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, sixth means for obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and seventh means for synthesizing the projected image of the object at a position corresponding to the fourth coordinates of the second image.
The image synthesizing method of comprises the first step of obtaining a first characteristic point nearby a synthesis position,for synthesizing an object among the characteristic points projected onto a first image, the second step of obtaining first coordinates on the three-dimensional coordinate system of a first characteristic point corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image, the fourth step of obtaining coordinate transform functions of the first coordinates and the second coordinates, the fifth step of obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, the sixth step of obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the seventh step of synthesizing the projected image of an object at a position corresponding to the fourth coordinates of the second image.
The supply medium of provides a computer program used for an image synthesizing apparatus for synthesizing the projected image,of an object with at least first and second images and comprising the first step of obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto the first image, the second step of obtaining first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, the fourth step of obtaining coordinate transform functions of the first and second coordinates, the fifth step of obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, the sixth step of obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the seventh step of synthesizing the projected image of an object at a position corresponding to the fourth coordinates of the second image.
The position detecting apparatus comprises first means for obtaining a first characteristic point nearby an object among the characteristic points projected onto a first image, second means for obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means for obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, fifth means for obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, sixth means for obtaining coordinate transform functions of the first and second coordinates, seventh means for obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and eighth means for detecting the difference between the fourth coordinates and the fifth coordinates.
The position detecting method comprises the first step of obtaining a first characteristic point nearby an object among the characteristic points projected onto a first image, the second step of obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, the fourth step of obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, the fifth step of obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, the sixth step of obtaining coordinate transform functions of the first and second coordinates, the seventh step of obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the eighth step of detecting the difference between the fourth and fifth coordinates.
The supply medium provides a computer program used for a position detecting apparatus for detecting the positional relation of an object projected onto first and second images and comprising the first step of obtaining a first characteristic point nearby the object among the characteristic points projected onto the first image, the second step of obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, the fourth step of obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, the fifth step of obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, the sixth step of obtaining coordinate transform functions of the first and second coordinates, the seventh step of obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the eighth step of detecting the difference between the fourth and fifth coordinates.
The image synthesizing apparatus of claim 1 comprises image-pickup-device-position computing means for computing the most-probable position of an image pickup device when photographing each of a plurality of photographic images, characteristic point position computing means for computing the most-probable position of the characteristic point of a first object, virtual projected position computing means for computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, distortion value computing means for computing a distortion value in accordance with the difference between a virtual projected position and the position of a projected image of a characteristic point actually photographed on each of a plurality of photographic images, correcting means for correcting a position to which the projected image of a second object is set in accordance with the distortion value computed by the distortion value computing means, projected image computing means for computing the projected image of the second object when photographing the second object from an image pickup device at the most-probable position by assuming that the second object is located at the corrected position by the correcting means, and synthesizing means for synthesizing the projected image of the second object computed by the projected image computing means with each photographic image.
The image synthesizing method of claim 3 comprises the image-pickup-device-position computing step of computing the most-probable position of an image pickup device when photographing each of a plurality of photographic images, the characteristic point position computing step of computing the most-probable position of the characteristic point of a first object, the virtual projected position computing step of computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, distortion value computing step of computing a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of a characteristic point actually photographed on each of a plurality of photographic images, the correcting step of correcting a position to which the projected image of a second object is set in accordance with the distortion value computed in the distortion value computing step, the projected image computing step of computing the projected image of the second object when photographing the second object from an image pickup device at the most-probable position by assuming that the second object is located at the position corrected in the correcting step, and the synthesizing step of synthesizing the projected image of the second object computed in the projected image computing step with each photographic image.
The supply medium of claim 5 provides a computer program used for an image synthesizing apparatus for synthesizing the projected image of a second object with each of a plurality of photographic images on which a first object photographed by an image pickup device is photographed by assuming that the second object is located at a predetermined position and comprising the image-pickup-device-position computing step of computing the most-probable position of the image pickup device when photographing each of a plurality of photographic images, the characteristic point position computing step of computing the most-probable position of the characteristic point of the first object, virtual projected position computing step of computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, the distortion value computing step of computing a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images, the correcting step of correcting a position to which the projected image of a second object is set in accordance with the distortion value computed in the distortion value computing step, projected image computing step of computing the projected image of the second object when photographing the second object from an image pickup device at the most-probable position by assuming that the second object is located at the position corrected in the correcting step, and the synthesizing step of synthesizing the projected image of the second object computed in the projected image computing step with each photographic image.
In the case of the image synthesizing apparatus, first means obtains a first characteristic point nearby a synthesis position for synthesizing the image of an object among the characteristic points projected onto a first image, second means obtains first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means obtains second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image, fourth means obtains coordinate transform functions of the first and second coordinates, fifth means obtains third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing the object with the first image, sixth means obtains fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and seventh means synthesizes the projected image of the object at a position corresponding to the fourth coordinates of the second image.
In the case of the image synthesizing method and the supply medium, a first characteristic point nearby a synthesis position for synthesizing an object is obtained among the characteristic points projected onto a first image, first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image are obtained, second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image are obtained, coordinate transform functions of the first and second coordinates are obtained, third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing the object with the first image are obtained, and fourth coordinates on a three-dimensional coordinate system corresponding to the second image are obtained by applying the coordinate transform functions to the third coordinates to synthesize the projected image of the object at a position corresponding to the fourth coordinates of the second image.
In the case of the position detecting apparatus, first means obtains a first characteristic point nearby an object among the characteristic points projected onto a first image, second means obtains first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means obtains second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means obtains third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, fifth means obtains fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, sixth means obtains coordinate transform functions of the first and second coordinates, seventh means obtains fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and eighth means detects the difference between the fourth and fifth coordinates.
In the case of the position detecting method and the supply medium, a first characteristic point nearby an object is obtained among the characteristic points projected onto a first image, first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image are obtained, second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image are obtained among the characteristic points projected onto the second image, third coordinates of the object on a three-dimensional coordinate system corresponding to the first image are obtained, fourth coordinates of the object on a three-dimensional coordinate, system corresponding to the second image are obtained, coordinate transform functions of the first and second coordinates are obtained, and fifth coordinates on a three-dimensional coordinate system corresponding to the second image are obtained by applying the coordinate transform functions to the third coordinates to detect the difference between the fourth and fifth coordinates.
In the case of the image synthesizing apparatus of claim 1, image-pickup-device-position computing means computes the most-probable position of an image pickup device when photographing each of a plurality of photographic images, characteristic point position computing means computes the most-probable position of the characteristic point of a first object, virtual projected position computing means computes a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, distortion value computing means computes a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images, correcting means corrects a position to which the projected image of a second object is set in accordance with the distortion value computed by the distortion value computing means, projected image computing means computes the projected image of the second object when photographing the second object from an image pickup device present at the most-probable position by assuming that a second object is located at the position corrected by the correcting means, and synthesizing means synthesizes the projected image of the second object computed by the projected image computing means with each photographic image.
In the case of the image synthesizing method of claim 3 and the supply medium of claim 5, the most-probable position of an image pickup device when photographing each of a plurality of photographic images is computed, the most-probable position of a characteristic point of a first object is computed, a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device set to the most-probable position is computed, a distortion value is computed in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images, a position to which the projected image of a second object is set in accordance with the computed distortion value is corrected, and the projected image of the second object when photographing the second object from an image pickup device present at the most-probable position is computed by assuming that the second object is located at a corrected position to synthesize the projected image of the computed second object with each photographic image.